The present invention relates to an image processing apparatus, which is carried on a movable body such as a vehicle or the like, for detecting or measuring a situation or the like around the movable body.
In recent years, there prevails an image processing apparatus for detecting a feature or situation of an object to be observed. Specifically, for example, an image processing apparatus has been generally known, which is carried on a vehicle, for processing images obtained by photographing a scene outside of the vehicle so as to obtain various information which is useful for driving or moving the vehicle (Hereinafter, the apparatus may be referred to xe2x80x9con-vehicle image processing apparatusxe2x80x9d.). In the on-vehicle image processing apparatus described above, images, which continuously change wholly as the vehicle moves, are processed so that various feature information, such as a situation of the road and affairs around the road or motion of other vehicles, is extracted.
Further, generally among the image processing apparatus in which any feature portion in the images is extracted, particularly in the image processing apparatus in which the motion of a specified feature portion of the images is also extracted, the motion of the feature portion is extracted by motion compensation of MPEG (Moving Picture Expert Group), for example. Hereupon, the xe2x80x9cmotion compensationxe2x80x9d means such an image processing procedure that divides each of the frames of the photographed images into a plurality of blocks, detects that a specified block (i.e. motion compensation block) including a feature portion in a certain frame is similar to another block in the preceding frame according to a value of correlation in a defined search area. Further, the motion of the feature portion among the frames is detected by detecting displacement of the center of gravity of the block in the search area (For example, see the magazine of xe2x80x9cElectronicsxe2x80x9d, March, 1996.).
Further, a specific example of a, conventional image processing apparatus for detecting motion of a movable body is disclosed, for example, in the Japanese Laid-open Patent Publication No. 6-337938.
As shown in FIG. 16, the image processing apparatus disclosed in the Japanese Laid-open Patent Publication No. 6-337938 is provided with (a) image input means for inputting images including objects and their background, (b) background image extracting means for holding and outputting the background, (c) first average background extracting means for extracting the images including stationary objects, moving objects which are moving below the first velocity, and the background, (d) second average background extracting means for extracting the images including stationary objects, moving objects which are moving below the second velocity, and the background, (e) first difference processing means for seeking the difference between the output of the background image extracting means and the output of the first average background extracting means so as to generate velocity images on the basis of the difference, (f) second difference processing means for seeking the difference between the outputs of the both average background extracting means so as to generate velocity images on the basis of the difference, (g) third difference processing means for seeking the difference between the output of the image input means and the output of the second average background extracting means so as to generate velocity images on the basis of the difference.
However, in the conventional image processing apparatus utilizing the motion compensation of the MPEG described above, there is such a problem that although is is possible to detect the motion of the moving object within the block obtained by dividing the frame, it is impossible to extract the moving object from all the images.
Meanwhile, in the conventional image processing apparatus disclosed in the Japanese Laid-open Patent Publication No. 6-337938, there is such a problem that although it is possible to extract the moving object when the background is fixed, it is impossible to extract the moving object when the background changes as in the case of a on-vehicle image processing apparatus. That is, in the case that the conventional image processing apparatus is carried on the vehicle, because the shape of the background images in front of the vehicle continuously changes, it is impossible to exactly extract the motion of the background so that it is impossible to extract an obstruction or the like which is approaching the present vehicle, for example.
Moreover, in the conventional on-vehicle image processing apparatus in which the background images continuously change, there is such a problem that because it is required to process a huge amount of image data, the construction of the image processing apparatus is large-sized or complicated.
The present invention has been developed to solve the conventional problems described above, and has an object of providing an image processing apparatus of simple construction carried on a movable body such as a vehicle, which can extract the moving object or the like from the images over the whole images, even if the background continuously changes.
According to a first aspect of the present invention which is achieved to solve the above-mentioned problems, there is provided (i) an image processing apparatus carried on a movable body (apparatus which can move, for example, vehicle such as automobile etc.), which includes (ii) an image pick-up means for continuously photographing a scene (for example, road and its circumstance etc.) outside of the movable body to continuously form an image (pick-up image) corresponding to the scene, (iii) a feature extracting means for processing the image formed by the image pick-up means to continuously extract a predetermined feature (feature information) in the scene (iv) a transmitting means for transmitting the feature extracted by the feature extracting means to an external device, and (v) a recording means for recording therein the feature extracted by the feature extracting means.
According to the image processing apparatus, even if the background scene continuously changes, the feature information such as a moving object can be extracted from all the images. Moreover, because the extracted feature information can be transmitted to the outer device and recorded, the feature information can be utilized for driving or moving the movable body.
In the image processing apparatus according to the first aspect of the present invention, it is preferable that the apparatus further includes (vi) a feature estimating means for estimating a feature which is to be subsequently extracted by the feature extracting means on the previously extracted feature which has been extracted by the feature extracting means or the feature being recorded in the recording means, (vii) wherein the feature extracting means estimates the next feature on the basis of the feature estimated by the feature estimating means.
In this case, because the synopsis of the next feature can be grasped based on the estimated feature, the next feature may be easily extracted so that the time required for extracting the feature may be reduced. Moreover, the data volume of the image processing apparatus may be small.
According to the second aspect of the present invention, there is provided (i) an image processing apparatus carried on a movable body, which includes (ii) an image pick-up means for continuously photographing a scene outside of the movable body to continuously form an image corresponding to the scene (iii) a feature extracting means for processing the image formed by the image pick-up means to continuously extract a predetermined feature in the scene (iv) an intermittent image getting means for intermittently getting a stationary image at a predetermined periodic interval from the image formed by the image pick-up means (v) a synthesizing means for synthesizing the feature extracted in the periodic interval by the feature extracting means and the stationary image obtained by the intermittent image getting means so as to form a whole image during the periodic interval (vi) a transmitting means for transmitting the image formed by the synthesizing means to an outer device, and (vii) a recording means for recording therein the image formed by the synthesizing means.
According to the image processing apparatus, the compressibility of the data can be elevated (improved) by intermittently compressing the image so as to record it, and further by extracting the feature data within the partial region in which the image exists and the interval region. Further, when the image is reproduced, the necessarily moving image is reconstructed from the image data and the feature data.
Therefore, even if the background scene continuously changes, the feature information such as a moving object can be extracted from the images over all the images. Further, because various feature information is obtained on the basis of the stationary image which is intermittently obtained, the amount of data may be extremely decreased so that the image processing apparatus may be simplified. Moreover, because the synthesized image or feature information can be transmitted to the outer device and recorded, the synthesized image or feature information can be utilized for driving or moving the movable body.
In the image processing apparatus according to the second aspect of the present invention, it is preferable that the intermittent image getting means changes (a) a procedure to get the stationary image, (b) compressibility of the image, (c) the periodic interval to get the image, or the like, according to kind or amount of change of the feature extracted by the feature extracting means. In this case, the amount of the image data may be further decreased so that the image processing apparatus may be further simplified.
In both of the image processing apparatuses described above, it is preferable that each apparatus includes a danger estimating means for estimating a dangerous situation which may be caused in the movable body on the basis of the feature extracted by the feature extracting means. In this case, because the dangerous situation which may be caused in the movable body can be estimated, the movable body can be safely driven or moved.
In both of the image processing apparatuses described above, the feature extracting means can process the image corresponding to the scene which moves wholly relative to the vehicle (i.e. image which relatively moves), or the image corresponding to the scene which does not move wholly relative to the vehicle (i.e. image which does not relatively move), so as to extract various features, each of the images being formed by the image pick-up means.
Hereupon, when the movable body is a vehicle (e.g. automobile) for example, the feature in the scene, which is to be extracted by the feature extracting means, may be (a) a moving object such as another vehicle or a pedestrian, (b) velocity of the center of gravity of the moving object, (c) outlines of the road and the traffic lane on which the vehicle is moving, (d) curvature radius of a winding road, (e) width of the traffic lane on which the vehicle is moving, (f) number of the traffic lanes of the road, (g) a road crossing, (h) the position of the vehicle within the road and the traffic lane on which the vehicle is moving, (i) whether both side road-indicating lines defining the road or the traffic lane, on which the vehicle is moving, are continuous lines or broken lines, (j) color of the both side road-indicating lines defining the Toad or the traffic lane, on which the vehicle is moving, (k) color of a traffic control sign, (l) shape of the traffic control sign, (m) a guide sign, (n) a signal, (o) signal color of the signal, (p) flashes of signal light of the signal, (q) a guardrail, (r) a side wall of the road, (s) a parked vehicle, (t) a traffic obstruction, (u) a barrier for a railroad crossing, or the like.